Display device performing screen mirroring and operating method thereof

ABSTRACT

A display device includes: a display, a communicator comprising communication circuitry configured to communicate with a mobile device, a memory storing one or more instructions, and a processor configured to execute the one or more instructions stored in the memory to: control the display to display a second screen corresponding to a first screen output on the mobile device and an indicator based on the display device mirroring a screen of the mobile device, control the communicator to receive object information regarding one or more objects included in the first screen from the mobile device, and to control movement of the indicator based on the object information and the control signal, in response to receipt of a control signal including interaction information regarding the indicator.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2019-0166004, filed on Dec. 12,2019, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a display device and an operating methodthereof, and for example, to a display device which displays a screenoutput on a mobile device by mirroring the screen and performs aninteraction with the displayed mirroring screen using a control device,and an operating method of the display device.

2. Description of Related Art

With the development of wired/wireless communication networks,electronic devices that output data visually perceivable by a user bydisplaying a screen may be interconnected via a wired/wirelesscommunication network.

The electronic devices may transmit and receive various data to and fromone another via the wired/wireless communication network, in which anelectronic device may remotely control another electronic device or maybe used using the other electronic device. For the above-describedremote control or shared use between electronic devices, a mirroringtechnique has been used.

The mirroring technique allows devices including displays to share andprocess screen data.

The mirroring technique has been developed to be applied to anyelectronic device capable of outputting a screen through a display suchas a portable computer like a laptop computer, a netbook, a tabletpersonal computer (PC), etc., a portable terminal like a smartphone or apersonal digital assistant (PDA), a television (TV), and so forth. Amirroring service enabling screen sharing and mutual remote control byapplying the mirroring technique to multiple electronic devices has beenprovided.

SUMMARY

Embodiments of the disclosure provide a display device which displays ascreen output on a connected mobile device by mirroring the screen andeasily performs interaction with the displayed mirroring screen using acontrol device, and an operating method of the display device.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description.

A display device according to an example embodiment of the disclosureincludes: a display, a communicator comprising communication circuitryconfigured to communicate with a mobile device, a memory storing one ormore instructions, which, when executed, cause a processor to beconfigured to: control the display to display a second screencorresponding to a first screen output on the mobile device and anindicator based on the display device mirroring a screen of the mobiledevice, control the communicator to receive object information regardingone or more objects included in the first screen from the mobile device,and control movement of the indicator based on the object informationand the control signal, in response to receipt of a control signalincluding user interaction information regarding the indicator.

The control signal according to an embodiment of the disclosure includesthe user interaction information for moving the indicator.

The object information according to an embodiment of the disclosureincludes at least one of position information of the one or moreobjects, information about an object that adjoins the one or moreobjects, timestamp information, or information regarding whether the oneor more objects are associated with critical movement.

The processor according to an embodiment of the disclosure is furtherconfigured to execute the one or more instructions to allocate a numberto each of the one or more objects included in the second screen andcontrol the display to display the allocated number.

The processor according to an embodiment of the disclosure is furtherconfigured to execute the one or more instructions to move the indicatorto an object to which the indicator is allocated, in response to receiptof a control signal including user interaction information to select thenumber.

The processor according to an embodiment of the disclosure is furtherconfigured to execute the one or more instructions to generatenavigation map information of the one or more objects based on theobject information and control movement of the indicator based on thenavigation map information.

The processor according to an embodiment of the disclosure is furtherconfigured to execute the one or more instructions to determine based onthe object information and the navigation map information whether thecontrol signal includes user interaction information regarding an objectassociated with critical movement, and control the communicator totransmit the user interaction information regarding the object to themobile device when the control signal includes the user interactioninformation regarding the object associated with critical movement.

The user interaction information regarding the object according to anembodiment of the disclosure includes user interaction information tomove the indicator to an object that causes a page changeover operationamong the one or more objects.

The processor according to an embodiment of the disclosure is furtherconfigured to execute the one or more instructions to, based on a screenof the mobile device being changed from the first screen to a thirdscreen as the mobile device performs an operation corresponding to theuser interaction information regarding the object, control the displayto display a fourth screen corresponding to the third screen bymirroring the third screen.

The processor according to an embodiment of the disclosure is furtherconfigured to execute the one or more instructions to perform anoperation corresponding to the user interaction information regardingthe object when the control signal does not include the user interactioninformation regarding the object associated with critical movement.

A method of operating a display device according to an embodiment of thedisclosure includes: displaying a second screen corresponding to a firstscreen output on a mobile device and an indicator based on the displaydevice mirroring a screen of the mobile device, receiving objectinformation regarding one or more objects included in the first screenfrom the mobile device, receiving a control signal including userinteraction information regarding the indicator, and controllingmovement of the indicator based on the object information and thecontrol signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing detailed description, taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a diagram illustrating an example mirroring system accordingto various embodiments;

FIG. 2 is a flowchart illustrating an example method of operating adisplay device according to various embodiments;

FIG. 3 is a diagram illustrating an example of object informationaccording to various embodiments;

FIG. 4 is a diagram illustrating example navigation map informationaccording to various embodiments;

FIG. 5 is a flowchart illustrating an example method of operating adisplay device according to various embodiments;

FIGS. 6A, 6B and 6C are diagrams illustrating an example in which anindicator displayed on a display device moves based on a userinteraction, in the display device according to various embodiments;

FIG. 7 is a signal flow diagram illustrating an example method ofoperating a mirroring system according to various embodiments;

FIG. 8 is a diagram illustrating an example in which a display devicedisplays a second screen according to various embodiments;

FIG. 9 is a diagram illustrating an example in which a display devicedisplays a second screen according to various embodiments;

FIGS. 10A, 10B, 10C and 10D are diagrams illustrating an example inwhich an indicator displayed on a display device a moves based on a userinteraction, in the display device according to various embodiments;

FIGS. 11A and 11B are diagrams illustrating an example in which adisplay device mirrors a screen of a mobile device according to variousembodiments;

FIGS. 12A, 12B, 12C and 12D are diagrams illustrating an example inwhich an indicator displayed on a display device moves based on a userinteraction, in the display device according to various embodiments;

FIG. 13 is a block diagram illustrating an example configuration of adisplay device according to various embodiments; and

FIG. 14 is a block diagram illustrating an example configuration of adisplay device according to various embodiments.

DETAILED DESCRIPTION

Terms used herein will be described in brief, and the disclosure will bedescribed in greater detail.

Although terms used in the disclosure are selected with general termsused at present in consideration of functions in the disclosure, theterms may vary according to the intention of those of ordinary skill inthe art, judicial precedents, or introduction of new technology. Inaddition, in some cases, terms may be arbitrarily selected, and in suchcases, the meaning of the terms is disclosed in a correspondingdescription part of the disclosure. Thus, the terms used in thedisclosure should be defined not by the simple names of the terms but bythe meaning of the terms and the contents throughout the disclosure.

Throughout the entirety of the specification of the disclosure, when itis assumed that a certain part “includes” a certain component, acorresponding component may further include other components unless aspecific meaning opposed to the corresponding component is written. Theterm used in the embodiments of the disclosure such as “unit” or“module” indicates a unit for processing at least one function oroperation, and may be implemented in hardware, software, or in acombination of hardware and software.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

In an embodiment of the specification, the term “user” may refer, forexample, to a person who controls a function or an operation of a homeappliance (or a peripheral device), a display device, and a mobiledevice, and may include a manager or an installation engineer.

Hereinafter, various example embodiments will be described in greaterdetail with reference to the attached drawings. However, the disclosuremay be implemented in various forms, and are not limited to theembodiments described herein. To clearly describe the disclosure, partsthat are not associated with the description may be omitted from thedrawings, and throughout the disclosure, identical reference numeralsrefer to identical parts.

FIG. 1 is a diagram illustrating an example mirroring system accordingto various embodiments.

Referring to FIG. 1, a mirroring system according to an embodiment mayinclude an electronic device 200 and a display device 100. Theelectronic device 200 and the display device 100 according to anembodiment may be devices providing a mirroring function. The mirroringfunction may refer, for example, to a function that enables devicesincluding displays to share and process screens (or screen data).

To provide the mirroring function, a device for providing an image (or ascreen) may transmit data regarding the image to be shared to a devicefor being provided with the image. The device for providing the imagemay be referred to as a source device or a master device, and the devicebeing provided with the image may be referred to as a sink device or aclient device. For example, in FIG. 1, the electronic device 200according to an embodiment may be a source device or a master device andthe display device 100 according to an embodiment may be referred to asa sink device or a client device.

The electronic device 200 according to an embodiment may be anyelectronic device capable of providing a mirroring function by sharingand displaying a screen. For example, the electronic device 200 mayinclude, but is not limited to, electronic devices capable of outputtinga screen through a display such as a portable computer like a laptopcomputer, a netbook, a tablet PC, etc., a portable terminal like asmartphone or a PDA, a TV, and so forth. In an embodiment, theelectronic device 200 operating as a source device in screen mirroringmay be an electronic device having a portable form, and hereinafter, theelectronic device 200 operating as a source device will be referred toas a ‘mobile device’ 200.

The display device 100 according to an embodiment, which may include anelectronic device including a display, may be implemented in variousforms such as, for example, and without limitation, a cellular phone, atablet PC, a digital camera, a camcorder, a laptop computer, a tabletPC, a desktop, an electronic (e)-book terminal, a digital broadcastingterminal, a PDA, a PMP, a navigator, an MP3 player, a wearable device,and so forth. The display device 100 may be of a fixed-type electronicdevice arranged at a fixed position or a portable-type electronic devicehaving a portable form, and may be a digital broadcasting receivercapable of receiving digital broadcasting. In particular, embodimentsmay be easily implemented in, but is not limited to, a TV, a digital TV,a smart TV, digital signage, a digital signal, etc., capable ofproviding a screen having a larger size than a screen of the mobiledevice 200.

The user may use the mirroring function between the mobile device 200and the display device 100 to watch an image output on the mobile device200 through a large screen. In this case, the mobile device 200 and thedisplay device 100 may be connected via a communication network. Thecommunication network connecting the mobile device 200 with the displaydevice 100 may include a wired/wireless communication network capable oftransmitting and receiving certain data. For example, a communicationnetwork may be formed according to wireless network standards such as alocal area network (LAN), a wide area network (WAN), a wireless LAN(WLAN) (Wireless Fidelity (WiFi), WiFi Direct, etc.), wireless broadband(WiBro), code division multiple access (CDMA), wireless CDMA (WCDMA),short range communication (near field communication (NFC)), Bluetooth,etc. The communication network may be a wired network such as a highdefinition multimedia interface (HDMI) cable, a mobile high definitionlink (MHL) cable, etc., which connects the mobile device 200 with thedisplay device 100 via a wired cable.

When the mirroring function is used, the screen output through thedisplay of the mobile device 200 may be mirrored and the mirrored screenmay be output through the display of the display device 100. In thiscase, the screen may refer, for example, to an image output through thedisplay or a region of the display where the image is displayed.

The mirroring technique used to provide the mirroring function mayinclude, but is not limited to, a display as a service (Daas), miracast,airplay, digital living network allicane (DLNA), etc. In an embodiment,to transmit information or data regarding a screen corresponding to thescreen output on the mobile device 200 to the display device 100, theabove-described mirroring techniques may be used.

Referring to FIG. 1, the mobile device 200 according to an embodimentmay display a first screen 10. For example, the first screen 10 may be,but is not limited to, a user interface screen including one or moreobjects.

Upon execution of the mirroring function, the display device 100according to an embodiment may mirror the first screen 10 output on themobile device 200 to display a second screen 20 corresponding to thefirst screen 10 on a display. The second screen 20 may be displayed onthe entire region or a partial region of the display. The second screen20 may be all or a part of the first screen 10 and may be an enlarged orreduced screen of the first screen 10.

The display device 100 may have an advantage of a “lean-back experience”due to characteristics of the display device 100 having a large screen.Herein, the “lean-back experience” may refer, for example, to a userexperience (UX) to which simplicity and comfort are key as if the userenjoys contents on a large screen by merely holding a remote controller(control device) of a display device while leaning back against a largesofa like when lying.

To manipulate a screen without hindering a viewing environment of theuser watching a mirroring screen (second screen), there is a need for amethod of manipulating the second screen 20 using a control device 50that controls the display device 100 instead of controlling the firstscreen 10 of the mobile device 200.

The control device 50 according to an embodiment may be implemented withvarious forms of devices for controlling the display device 100, like aremote controller or a cellular phone. Alternatively, the control device50 may control the display device 100 using short-range communicationincluding infrared communication or Bluetooth. The control device 50 maycontrol a function of the display device 100 using at least one of a key(including a button), a touch pad, a microphone (not shown) capable ofreceiving a user's voice, or a sensor (not shown) capable of recognizinga motion of the control device 50, which is provided therein.

Hereinafter, by referring to the drawings, an example method ofmanipulating a mirroring screen displayed on the display device 100using the control device 50 will be described in greater detail.

FIG. 2 is a flowchart illustrating an example method of operating adisplay device according to various embodiments.

Referring to FIG. 2, the display device 100 according to an embodimentmay mirror a first screen output on the mobile device 200 connected withthe display device 100. The display device 100 may receive mirroringdata corresponding to the first screen from the mobile device 200.

The display device 100 may display a second screen corresponding to thefirst screen based on the mirroring data, and display an indicator onthe second screen, in operation S210.

In this case, the display device 200 may display the second screen onthe entire region or a partial region of the display. The second screenmay be all or a part of the first screen and may be an enlarged orreduced screen of the first screen.

The display device 100 may receive object data (information)corresponding to the first screen from the mobile device 200, inoperation S220.

The display device 100 may receive the object information together whenreceiving the mirroring data of the first screen, or may request theobject information to and receive the object information from the mobiledevice 200 when receiving a control signal including user interactioninformation (e.g., a user input of four-direction keys of the controldevice) from the control device. Upon receipt of a user input forrequesting mirroring in the mobile device 200, the object informationmay be transmitted to the display device 100. However, the disclosure isnot limited to this example. The object information will be describedwith reference to FIG. 3.

FIG. 3 is a diagram illustrating an example of object informationaccording various embodiments.

The mobile device 200 according to an embodiment may obtain objectinformation about one or more objects included in the first screen 10output on a display of the mobile device 200. In this case, the one ormore objects included in the first screen may be user interface objectswhich may be targets for a user interaction. For example, the objectinformation may include an object map 310 and an object table 320.

The object map 310 may indicate coordinate information, boundaryinformation, etc., of objects included in the first screen 10, as shownin FIG. 3. As shown in FIG. 3, the object table 320 may include, but isnot limited to, identification information of an object (e.g., an objectidentifier (ID)), position information of the object (e.g., coordinateinformation), timestamp information of the object, information aboutwhether the object is associated with critical movement, etc.

The mobile device 200 according to an embodiment may obtain objectinformation by analyzing a framework of an application execution screenwhen the first screen 10 being currently output on the mobile device 200is the application execution screen. The mobile device 200 may obtainthe object information by analyzing rendering information, etc.,regarding a web page, when the first screen 10 is a web page screen. Themobile device 200 may obtain the object information through computervision analysis of the first screen 10 being currently output, etc.However, the disclosure is not limited thereto, and the mobile device 10may obtain the object information in various manners.

The mobile device 200 may determine whether the object is associatedwith critical movement. For example, when certain contents are executedor a page is moved or changed through a user interaction with theobject, the object may be determined to be associated with criticalmovement. For example, when the certain contents corresponding to theobject are executed through a user interaction to select the object, theobject may be determined to be associated with critical movement withrespect to a user's selection input. When a page changeover operation ora screen changeover operation occurs through a user interaction to movean indicator with an object located on a boundary of the screen, theobject may be determined to be associated with critical movement.

On the other hand, as a user interaction to move an indicator betweenobjects does not cause a content execution operation, a page changeoveroperation, etc., a corresponding object may be determined not to beassociated with critical movement.

Timestamp information of an object according to an embodiment may refer,for example, to information about a time at which the object isdisplayed on a screen. For example, objects displayed on an identicalscreen may include identical timestamp information. The timestampinformation of an object may also be used to identify an object forwhich an interaction occurs and to determine whether the interaction forthe object is valid, between the mobile device 200 (a source device) andthe display device 100 (a sink device).

The mobile device 200 according to an embodiment may transmit theobtained object information to the display device 100 that is a sinkdevice. For example, upon execution of the mirroring function, themobile device 200 may transmit object information regarding the firstscreen being currently output on the mobile device 200 to the displaydevice 100. The mobile device 200 may transmit the object informationregarding the first screen, together with mirroring information(mirroring data) regarding the first screen.

Upon receipt of an object information request from the display device100, the mobile device 200 may transmit the object information regardingthe first screen to the display device 100.

Referring back to FIG. 2, the display device 100 may receive a controlsignal for moving the indicator displayed on the mirroring screen (thesecond screen) in operation S230. For example, to move the indicator,the user may perform an input to press any one of four direction keys ofthe control device 50, an input to drag in a particular direction on atouch pad of the control device 50, an action to incline or move thecontrol device 50 in a particular direction, etc., and the controldevice 50 may transmit a control signal including user interactioninformation to the display device 100.

The display device 100 according to an embodiment may control movementof the indicator based on the object information and the receivedcontrol signal from the control device, in operation S240.

The display device 100 may obtain navigation map information based onthe object information received from the mobile device 200. Thenavigation map information will be described in greater detail belowwith reference to FIG. 4.

FIG. 4 is a diagram illustrating an example navigation map informationaccording to various embodiments.

Referring to FIG. 4, navigation map information 410 may include a mapdefining a target object regarding a four-direction input of each ofobjects included in the second screen. For example, the navigation mapinformation 410 may define that when the indicator is located in thefirst object 420, with respect to a first object 420, upon receipt of anupward input, the indicator may be moved to a second object 430; uponreceipt of a downward input, the indicator may be moved to a thirdobject 440; upon receipt of a left input, the indicator may be moved toa fourth object 450; and upon receipt of a right input, the indicatormay be moved to a fifth object 460. In the same manner, for each ofobjects included in the second screen, a target object to which anindicator is to be moved in response to a four-direction input may bedefined. The navigation map information 410 may also be referred to asfour-direction map information.

The display device 100 according to an embodiment may control movementof the indicator between the objects based on the navigation mapinformation, upon receipt of the four-direction input from the controldevice.

FIG. 5 is a flowchart illustrating an example method of operating adisplay device according to various embodiments.

Referring to FIG. 5, the display device 100 according to an embodimentmay mirror the first screen output on the mobile device 200 to displaythe second screen corresponding to the first screen and may display theindicator on the second screen, in operation S510. Operation S510 maycorrespond to operation S210 of FIG. 2, and thus a detailed descriptionthereof may not be repeated here.

The display device 100 according to an embodiment may receive a controlsignal including user interaction information regarding the indicator,in operation S520.

For example, the user interaction information may include, but is notlimited to, a user input to move the indicator, an input to select anobject on which the indicator is located, etc.

The display device 100 according to an embodiment may determine whetheran operation corresponding to a user interaction included in the controlsignal causes critical movement, in operation S530.

For example, as a user interaction to select an object on which theindicator is currently located causes an operation in which particularcontents corresponding to the object are executed, the user interactionmay be determined to cause critical movement. A user interaction to movethe indicator to an object located in a boundary of the screen may causethe page changeover operation, and thus may be determined to causecritical movement. On the other hand, a user interaction to move theindicator to an object not located in the boundary of the screen maymove the indicator without causing the page changeover operation, andthus may be determined not to cause critical movement. However, thedisclosure is not limited to this example.

When an operation corresponding to the user interaction does not causecritical movement (“No” in operation S530), the display device 100 mayperform the operation corresponding to the user interaction, inoperation S540.

For example, for a user interaction to move the indicator from a firstobject on which the indicator is currently located to a second objectnot located in the boundary of the screen, the display device 100 maymove the indicator from the first object to the second object anddisplay the indicator on the second object.

On the other hand, when an operation corresponding to the userinteraction causes critical movement (“Yes” in operation S530), thedisplay device 100 may transmit user interaction information to themobile device 200, in operation S550.

For example, for a user interaction to move the indicator from the firstobject on which the indicator is currently located to a third objectlocated in the boundary of the screen, the display device 100 maytransmit user interaction information for moving the indicator to thethird object to the mobile device 200. The display device 100 maytransmit timestamp information regarding the third object to the mobiledevice 200. The mobile device 200 may perform a page changeoveroperation corresponding to a user input to move the indicator to thethird object, based on the received user interaction information. Forexample, the mobile device 200 may search an object map or an objecttable based on timestamp information and coordinate informationregarding the third object, thereby identifying the third object. Themobile device 200 may compare the timestamp information of the thirdobject with timestamp information of a mirroring screen being currentlytransmitted to the display device 100 and determine whether the userinput to move the indicator to the third object is valid. When the thirdobject is not included in the mirroring screen being currentlytransmitted to the display device 100, the mobile device 200 may notperform an operation corresponding to the interaction (e.g., the pagechangeover operation).

On the other hand, when the third object is included in the mirroringscreen being currently transmitted to the display device 100, the mobiledevice 200 may perform the operation corresponding to the interaction(e.g., the page changeover operation).

The mobile device 200 may output a third screen including the changedpage on the display of the mobile device 200 and transmit mirroring datacorresponding to the third screen to the display device 100. The displaydevice 100 may receive the mirroring data in operation S560, and outputa fourth screen corresponding to the third screen and display thechanged page on the fourth screen based on the mirroring information inoperation S570.

When the display device 100 according to an embodiment receives thecontrol signal including the user interaction information in operationS520, the display device 100 may identify a connection state of anetwork connecting the display device 100 with the mobile device 200 andmay determine whether the operation corresponding to the userinteraction causes critical movement when the connection state of thenetwork is not good. On the other hand, when the connection state of thenetwork is good, the display device 100 may transmit the userinteraction information to the mobile device 200 regardless of whetherthe operation corresponding to the user interaction causes criticalmovement.

The display device 100 according to an embodiment may identify a graphicprocessing unit (GPU) resource state thereof, and when the amount of GPUresources is greater than or equal to a preset value, the display device100 may determine whether the operation corresponding to the userinteraction causes critical movement and determine whether to transmitthe user interaction information to the mobile device 200 based onwhether the operation causes critical movement. On the other hand, whenthe amount of GPU resources is less than the preset value, the displaydevice 100 may transmit the user interaction information to the mobiledevice 200 regardless of whether the operation corresponding to the userinteraction causes critical movement.

FIGS. 6A, 6B and 6C are diagrams illustrating an example in which anindicator displayed on a display device moves based on a userinteraction, in the display device according to various embodiments.

Referring to FIG. 6A, the display device 100 according to an embodimentmay mirror a first screen 610 of the mobile device 200 to display asecond screen 620 corresponding to the first screen 610.

The second screen 620 may include one or more objects, and the displaydevice 100 may receive object information regarding the one or moreobjects from the mobile device 200. The object information has beendescribed in detail with reference to FIG. 3 and thus may not bedescribed in detail here.

In addition, the display device 100 may display an indicator 630 on thesecond screen 620. The display device 100 may display the second screen620 on the entire region or a smaller region of the display.

When mirroring between the mobile device 200 and the display device 100starts, the indicator 630 may be displayed in a preset position. Forexample, an indicator may be displayed on an object positioned first ina left top end of the second screen 620, or the indicator may bedisplayed in the center of the second screen 620.

The indicator 630 according to an embodiment may, for example, be in theform of a focus or a pointer, and in FIGS. 6A, 6B and 6C, the indicator630 will be described as being in the form of a focus.

For example, as shown in FIG. 6A, when the indicator 630 is in the formof a focus, an object on which the focus 630 is positioned (a focusedobject) may be highlighted or an edge of the object on which the focus630 is positioned may be marked bold. However, the disclosure is notlimited thereto, and the display device 100 may be displayed to bedistinguished from other objects.

The display device 100 may generate navigation map information based onobject information, in which the navigation map information has beendescribed in detail with reference to FIG. 4 and thus may not bedescribed in detail here.

When the display device 100 receives a four-direction input (e.g., afour-direction key input of the control device 50) through the controldevice 50, the display device 100 may move the focus 630 (change thefocused object) based on the navigation map information.

For example, when the display device 100 receives the four-direction keyinput (e.g., up, down, left, and right direction key inputs) of thecontrol device 50, the display device 100 may move the focus from thecurrently focused object (object on which the focus is positioned) to anobject that adjoins the currently focused object above, under, to theleft of, or to the right of the currently focused object.

As shown in FIG. 6A, the display device 100 may receive a control signalcorresponding to a user input to press a down key among four-directionkeys of the control device 50 from the control device 50.

The display device 100 may identify a target object (a second object642) to which the focus 630 is to be moved, corresponding to a downinput of the currently focused object (a first object 641) based on thenavigation map information, and determine based on the objectinformation whether focus movement to the second object 642 causescritical movement.

When focus movement to the second object 642 does not cause criticalmovement, the display device 100 may move the focus 630 to the secondobject 642 based on the navigation map information as shown in FIG. 6B.

As shown in FIG. 6B, the display device 100 may receive a control signalcorresponding to a user input to press the down key among four-directionkeys of the control device 50 from the control device 50 in a statewhere the focus 630 is positioned on the second object 642 (the secondobject 642 is focused).

The display device 100 may identify a target object (a third object 643)to which the focus 630 is to be moved, corresponding to a down input ofthe currently focused second object 642 based on the navigation mapinformation, and determine based on the object information whether focusmovement to the third object 643 causes critical movement.

For example, the third object 643 may be positioned in a lower boundaryof the second screen 620, a part of the third object 643 may bedisplayed on the second screen 620, and the other part of the thirdobject 643 and objects below the third object 643 may not be displayedon the second screen 620. Thus, when the focus 630 is moved to the thirdobject 643, the page changeover operation may occur such that theobjects below the third object 643 may be displayed.

When focus movement to the third object 643 causes critical movement,the display device 100 may transmit user interaction informationincluded in the control signal to the mobile device 200.

Referring to FIG. 6C, the mobile device 200 may perform an operationcorresponding to a user interaction based on the user interactioninformation received from the display device 100. For example, themobile device 200 may perform the page changeover operationcorresponding to a user input to move the focus 630 to the third object643.

The mobile device 200 may output a third screen 615 including thechanged page on the display of the mobile device 200 and transmitmirroring data corresponding to the third screen 615 to the displaydevice 100. The display device 100 may display a fourth screen 625corresponding to the third screen 615 based on the received mirroringdata. The fourth screen 625 may include the changed page and the displaydevice 100 may position the focus 630 on the third object 643.

FIG. 7 is a signal flow diagram illustrating an example method ofoperating a mirroring system according to various embodiments.

Referring to FIG. 7, a mirroring system according to an embodiment mayinclude the mobile device 200, the display device 100, and the controldevice 50, and the display device 100 may mirror a screen output on themobile device 200.

Referring to FIG. 7, the mobile device 200 according to an embodimentmay display a first screen in operation S710. When mirroring starts inoperation S720, the mobile device 200 may transmit mirroring information(data) of the first screen to the display device 100. The display device100 may display a second screen corresponding to the first screen basedon the mirroring information of the first screen, and display anindicator on the second screen, in operation S730.

The mobile device 200 according to an embodiment may obtain objectinformation corresponding to the first screen, in operation S725. Themobile device 200 may obtain object information about one or moreobjects included in the first screen. The object information has beendescribed in detail with reference to FIG. 3 and thus may not bedescribed in detail here.

The mobile device 200 may transmit the obtained object information tothe display device 100 in operation S735.

The control device 50 according to an embodiment may receive a userinput (user interaction information) in operation S740. For example, theuser input may be an input to press four-direction keys included in thecontrol device 50, an input to incline or move the control device 50 ina certain direction, an input to drag on a touch pad included in thecontrol device 50 using a finger, etc., and so forth, without beinglimited thereto.

Upon receipt of the user input, the control device 50 may transmit acontrol signal including user interaction information to the displaydevice 100 in operation S745.

The display device 100 may determine whether an operation correspondingto a user interaction included in the control signal requires screenchangeover, in operation S750.

For example, in response to a user interaction to select an object onwhich the indicator is currently located, particular contentscorresponding to the object need to be executed and a content executionscreen needs to be displayed, such that the user interaction may bedetermined to be an operation requiring screen changeover. In addition,in response to a user interaction to move an indicator to an objectlocated in a boundary of the screen, a page is changed and the screenincluding the changed page needs to be displayed, such that the userinteraction may be determined to be an operation requiring screenchangeover. On the other hand, in response to a user interaction to movean indicator to an object not located in the boundary of the screen, thepage is not changed and thus the indicator is moved and displayedwithout screen changeover, such that the user interaction may bedetermined to be an operation not requiring screen changeover.

When an operation corresponding to the user interaction does not requirescreen changeover (“No” in operation S750), the display device 100 mayperform the operation corresponding to the user interaction, inoperation S760. In this case, even when the operation corresponding theuser interaction is performed, screen changeover does not occur.

When the operation corresponding to the user interaction requires screenchangeover (“Yes” in operation S750), the display device 100 maytransmit user interaction information to the mobile device 200, inoperation S765.

The mobile device 200 may perform the operation corresponding to theuser interaction in operation S770, such that a screen displayed on themobile device 200 may be changed from the first screen to the thirdscreen in operation S780.

The mobile device 200 may transmit mirroring information (data)corresponding to the third screen to the display device 100 in operationS785, and the display device 100 may display a fourth screencorresponding to the third screen based on the mirroring data inoperation S790.

FIG. 8 is a diagram illustrating an example in which a display devicedisplays a second screen according to various embodiments.

Referring to FIG. 8, the display device 100 according to an embodimentmay mirror a first screen 810 of the mobile device 200 to display asecond screen 820 corresponding to the first screen 810. The secondscreen 820 may include one or more objects, and the display device 100may receive object information regarding the one or more objects fromthe mobile device 200. The object information has been described indetail with reference to FIG. 3 and thus may not be described in detailhere.

As shown in FIG. 8, the display device 100 may allocate numbers toselectable objects included in the second screen 820 based on the objectinformation, and display the allocated numbers on the correspondingobjects.

When the user inputs a number using a numeric key of the control device50, the display device 100 may move an indicator to an objectcorresponding to the number. For example, when the user inputs a keycorresponding to a number ‘3’ of the control device 50, the displaydevice 100 may move an indicator 830 to an object 843 on which thenumber ‘3’ is displayed.

When the user inputs a number using a numeric key of the control device50, the display device 100 may control contents corresponding to anobject corresponding to a number to be executed. For example, when theuser inputs a key corresponding to a number ‘6’ of the control device50, the display device 100 may identify an object 846 on which thenumber ‘6’ is displayed and transmit a request for executing contentscorresponding to the object 846 to the mobile device 200. The mobiledevice 200 may then execute the contents corresponding to the object 846on which the number ‘6’ is displayed, and display a content executionscreen. The display 100 may mirror the content execution screen of themobile device 200 and display the content execution screen on thedisplay device 100.

FIG. 9 is a diagram illustrating an example in which a display devicedisplays a second screen according to various embodiments.

Referring to FIG. 9, the display device 100 according to an embodimentmay mirror a first screen 910 of the mobile device 200 to display asecond screen 920 corresponding to the first screen 910. The secondscreen 920 may include one or more objects, and the display device 100may receive object information regarding the one or more objects fromthe mobile device 200. The object information has been described indetail with reference to FIG. 3 and thus may not be described in detailhere.

As shown in FIG. 9, the display device 100 may generate an object list930 of selectable objects included in the second screen 920 based on theobject information and display the object list 930.

Referring to FIG. 9, the object list 930 according to an embodiment maybe a list of objects movable in two directions. For example, as shown inFIG. 9, the object list 930 may be a list of objects movable in up anddown directions. The object list 930 may be a list of objects movable inleft and right directions.

The object list 930 may include items respectively corresponding toselectable objects included in the second screen 920 and an indicator(e.g., a focus). A focus 940 displayed on the object list 930 may movebetween items included in the object list 930 based on a direction keyinput, a scroll key input, a wheel key input, etc., of the controldevice 50.

For example, upon receipt of an input of an up-direction key amongdirection keys of the control device 50, an input to scroll up a scrollkey, and an input to rotate a wheel key to the left, the display device100 may move the focus 940 up in the object list 930. For example, thedisplay device 100 may move the focus 940 from an item on which thefocus 940 is currently located to an item adjoining the item on whichthe focus 940 is currently located, above the item.

On the other hand, upon receipt of an input of a down-direction keyamong direction keys of the control device 50, an input to scroll down ascroll key, and an input to rotate a wheel key to the right, the displaydevice 100 may move the focus 940 down in the object list 930. Forexample, the display device 100 may move the focus 940 from the item onwhich the focus 940 is currently located to an item adjoining the itemon which the focus 940 is currently located, under the item.

In addition, the display device 100 may locate a focus 950 on an objectID #2 corresponding to an item ID #2 on which the focus 940 is located,in the object list 930 among objects included in the second screen 920.

When the user selects the item on which the focus 940 is located, usingthe control device 50, the display device 100 may identify an objectcorresponding to the item on which the focus 940 is located, andtransmit a request for executing contents corresponding to the object tothe mobile device 200. The mobile device 200 may then execute thecontents corresponding to the object, and display a content executionscreen. The display 100 may mirror the content execution screen of themobile device 200 and display the content execution screen on thedisplay device 100.

FIGS. 10A, 10B, 10C and 10D are diagrams illustrating an example inwhich an indicator displayed on a display device a moves based on a userinteraction, in the display device according to various embodiments.

Referring to FIG. 10A, the display device 100 according to an embodimentmay mirror a first screen 1010 of the mobile device 200 to display asecond screen 1020 corresponding to the first screen 1010. A secondscreen 1020 may be displayed on the entire region or a partial region ofthe display of the display device 100. The second screen 1020 mayinclude one or more objects, and the display device 100 may receiveobject information regarding the one or more objects from the mobiledevice 200. The object information has been described in detail withreference to FIG. 3 and thus may not be described in detail here.

In addition, the display device 100 may display an indicator 1030 on thesecond screen 1020. The indicator 1030 may be in the form of a focus ora pointer (or a cursor), and in FIGS. 10A, 10B, 100 and 10D, theindicator 1030 will be described, by way of non-limiting example, asbeing in the form of a pointer (or a cursor).

The control device 50 according to an embodiment may operate as apointing device. For example, the control device 50 may operate as apointing device upon receipt of a particular key input. When the controldevice 50 operates as a pointing device, the pointer 1030 may be movedbased on a user input to move the control device 50 up, down, to theleft, or to the right or incline the control device 50 in a certaindirection. For example, the control device 50 may detect movement of thecontrol device 50 and transmit movement information about the detectedmovement of the control device 50 to the display device 100. The displaydevice 100 may calculate coordinates of the pointer 1030 based on themovement information of the control device 50 and move the pointer 1030on the screen to correspond to the calculated coordinates.

The control device 50 according to an embodiment may include a touchpad, and in this case, the display device 100 may move the pointer 1030on the screen according to a displacement value of a subject such as auser's finger moving on the touch pad.

The control device 50 according to an embodiment may include fourdirection keys and may move the pointer 1030 on the screen in adirection and by a distance corresponding to a four-direction key input,based on the four-direction key input.

The display device 100 may determine selectable objects included in thesecond screen based on the object information. The display device 100may display a selectable object to be distinguished from anon-selectable object. For example, the display device 100 may mark anoutline of a selectable object with a dotted line or a bold line or maydisplay the selectable object in a different color, without beinglimited thereto. By displaying a selectable object distinguishably froma non-selectable object, the user may easily identify the selectableobject on the second screen.

The display device 100 may move the pointer 1030 based on objectinformation, upon receipt of a control signal including user interactioninformation from the control device 50. For example, when the userenters an input to move the pointer 1030 to the right using the controldevice 50, the display device 100 may move the pointer 1030 to the rightbased on a user input. When the pointer 1030 approaches a selectablesecond object 1042 as the pointer 1030 is moved to the right, thedisplay device 100 may determine based on the object information whetherpointer movement to the second object 1042 causes critical movement.

When pointer movement to the second object 1042 does not cause criticalmovement, the display device 100 may increase a moving speed of thepointer 1030 to move the pointer 1030 such that the pointer 1030 islocated on the selectable second object 1042, as shown in FIG. 10B. Forexample, when a distance between the pointer 1030 and the selectablesecond object 1042 is less than a threshold distance, the pointer 1030may be moved as if the pointer 1030 sticks to the center of theselectable second object 1042 like a magnet.

When the user continues entering the input to move the pointer 1030 tothe right using the control device 50 in a state where the pointer 1030is located in the center of the second object 1042, the display device100 may move the pointer 1030 to the right based on a user input, asshown in FIG. 100. When the pointer 1030 approaches a selectable thirdobject 1043 as the pointer 1030 is moved to the right, the displaydevice 100 may determine based on the object information whether pointermovement to the third object 1043 causes critical movement.

When pointer movement to the third object 1043 causes critical movement,the display device 100 may transmit user interaction informationincluded in the control signal to the mobile device 200.

The mobile device 200 may perform an operation corresponding to a userinteraction based on the user interaction information received from thedisplay device 100. For example, the mobile device 200 may output athird screen 1015 in which the page currently displayed on the firstscreen 1010 is changed to the next page as illustrated in FIG. 10D.

The display device 100 may output a fourth screen 1025 corresponding tothe third screen 1015 (a screen including the next page) displayed onthe mobile device 200 by mirroring the third screen 1015, and the fourthscreen 1025 may be a screen displaying the next page. The display device100 may locate the pointer 1030 on the third object 1043.

FIGS. 11A and 11B are diagrams illustrating an example in which adisplay device mirrors a screen of a mobile device according to variousembodiments.

Referring to FIG. 11A, the mobile device 200 according to an embodimentmay execute a first task and display a first window corresponding toexecution of the first task on a display of the mobile device 200. Forexample, the mobile device 200 may execute a first application anddisplay a first application execution window 1110 (hereinafter, a firstexecution window) on the display. Upon input of a request for mirroringwith the display device 100 during execution of the first task by themobile device 200, the display device 100 may display a screen 1120corresponding to the first execution window 1110 by mirroring the firstexecution window 1110. On the screen 1120 displayed on the display ofthe display device 100, an indicator 1130 may be displayed as shown inand described with reference to FIG. 6A. In FIG. 11A, the indicator 1130will be described by way of non-limiting example as being in a focusform.

The mobile device 200 according to an embodiment may performmulti-tasking. For example, the mobile device 200 may further perform asecond task while executing the first task. The mobile device 200 mayperform the most recently starting task in a foreground environment, andmay perform a previously performed task in a background environment. Forexample, the mobile device 200 may perform the first task in thebackground environment and the second task in the foregroundenvironment.

As shown in FIG. 11B, the mobile device 200 may output a second window1140 corresponding to the second task executed in the foregroundenvironment on the display and may output or may not output the firstwindow 1110 corresponding to the first task performed in the backgroundenvironment, in the form of a floating window on the display. FIG. 11Billustrates an example in which the first window 1110 corresponding tothe first task is not displayed on the display.

In this case, even when the mobile device 200 further performs thesecond task, the display device 100 may display the first window 1110corresponding to the first task continuously performed in the backgroundenvironment by mirroring the first window 1110.

The display device 100 may receive object information corresponding tothe first window 1110 from the mobile device 200. The display device 100may generate navigation map information based on object information, inwhich the navigation map information has been described in detail withreference to FIG. 4 and thus may not be described in detail here.

The display device 100 may receive a control signal including userinteraction information from the control device 50. For example, thedisplay device 100 may receive a four-direction key input of the controldevice 50, and may move the focus 1130 based on navigation mapinformation upon receipt of the four-direction key input.

For example, when the display device 100 receives the four-direction keyinput (e.g., up, down, left, and right direction key inputs), thedisplay device 100 may move the focus from the currently focused objectto an object that adjoins the currently focused object above, under, tothe left of, or to the right of the currently focused object.

The display device 100 may identify a target object to which the focus1130 is to be moved, based on navigation map information, and determinebased on the object information whether focus movement to the targetobject causes critical movement.

When focus movement to the target object does not cause criticalmovement, the display device 100 may move the focus 1130 to the targetobject based on the navigation map information.

On the other hand, when focus movement to the target object causescritical movement, the display device 100 may transmit user interactioninformation to the mobile device 200. The mobile device 200 may performan operation corresponding to a user interaction on a first windowcorresponding to a first task being currently executed in the backgroundenvironment, based on the user interaction information received from thedisplay device 100.

The mobile device 200 may transmit mirroring data of the first window1110 on which the operation corresponding to the user interaction isperformed to the display device 100 which may then display a screen onwhich the operation corresponding to the user interaction is performed,based on the mirroring data of the display device 100.

The mobile device 200 according to an embodiment may change a window tobe mirrored on the display device 100, upon receipt of a request forchanging a mirroring window. For example, upon receipt of a request formirroring the second window 1140 during mirroring of the first window1110 corresponding to the first task being executed in the backgroundenvironment, the second window 1140 corresponding to the second taskbeing currently executed in the foreground environment may be mirroredto the display device 100. The display device 100 may output a screencorresponding to the second window 1140 by mirroring the second window1140 of the mobile device 200.

FIGS. 12A, 12B, 12C and 12D are diagrams illustrating an example inwhich an indicator displayed on a display device moves based on a userinteraction, in the display device according to various embodiments.

Referring to FIG. 12A, the display of the display device 100 accordingto an embodiment may be divided into a first region 1210, a secondregion 1220, and a third region 1230. On the first region 1210,broadcasting contents being currently played on the display device 100may be output, without being limited thereto. On the second region 1220,a second screen mirroring a first screen of the mobile device 200 may bedisplayed. On the third region 1230, a menu list 1240 provided in thedisplay device 100 may be displayed. The menu list 1240 may include aplurality of menu items.

The display device 100 according to an embodiment may display anindicator 1250. The indicator 1250 may be in the form of a focus or apointer (or a cursor), and in FIGS. 12A, 12B, 12C and 12D, the indicator1250 will be described by way of non-limiting example as being in theform of a focus.

For example, as shown in FIG. 12A, the focus 1250 may be located on anyone of objects included in the second screen. The object on which thefocus 1250 is located may be highlighted or an outline of the object maybe displayed boldly, without being limited thereto, and the object onwhich the focus 1250 is located may be displayed distinguishably fromother objects.

The object on which the focus 1250 is located 1225 may be an objectlocated in the leftmost top end of the second screen. Thus, an objectadjoining the object above or to the left of the object may not exist.

The display device 100 may receive a control signal corresponding to auser input to press an up key or a left key among four-direction keys ofthe control device 50 from the control device 50. In the absence of anobject adjacent up or to the left of a currently focused object 1225,the display device 100 may control the focus 1250 to be located on theentire second region 1220 as shown in FIG. 12B.

Upon receipt of a signal to input a back button of the control device 50from the user, the display device 100 may control the focus 1250 to belocated on the entire second region 1220 regardless of existence orabsence of an object adjacent to the currently focused object.

When the focus 1250 is located on the entire second region 1220, thesecond region 1220 may be highlighted or an outline of the second region1220 may be displayed boldly, without being limited thereto.

When the focus 1250 is located on the second region 1220, focus movementmay occur among the first region 1210, the second region 1220, and thethird region 1230 through a four-direction key input of the controldevice 50. For example, upon receipt of a control signal to input a leftkey among four-direction keys of the control device 50 from the user ina state where the focus 1250 is located on the second region 1220, thedisplay device 100 may move the focus 1250 to the first region 1210.When the focus 1250 is moved to the first region 1210, the first region1210 may be highlighted or an outline of the first region 1210 may bedisplayed boldly.

For example, upon receipt of a control signal to input a down key amongthe four-direction keys of the control device 50 from the user in astate where the focus 1250 is located on the second region 1220, thedisplay device 100 may move the focus 1250 to the third region 1230.When the focus 1250 is moved to the third region 1230, the third region1230 may be highlighted or an outline of the third region 1230 may bedisplayed boldly as shown in FIG. 12C.

Upon receipt of a control signal to input a confirmation button of thecontrol device 50 from the user in a state where the focus 1250 islocated on the third region 1230, the display device 100 may positionthe focus 1250 on any one of a plurality of menu items included in themenu list 1240 displayed on the third region 1230, as shown in FIG. 12D.In this case, the focus 1250 may be located on a first menu item 1245which is located first among the plurality of menu items, without beinglimited thereto. When the focus 1250 is located on a menu item includedin the menu list 1240, focus movement may occur among menu items througha four-direction key input of the control device 50.

For example, upon receipt of a control signal to input a right key amongthe four-direction keys of the control device 50 from the user in astate where the focus 1250 is located on the first menu item 1245, thedisplay device 100 may move the focus 1250 to a second menu item 1247adjacent to the right of the first menu item 1245. When the focus 1250is moved to the second menu item 1247, the second menu item 1247 may behighlighted or an outline of the second menu item 1247 may be displayedboldly.

FIG. 13 is a block diagram illustrating an example configuration of adisplay device according to various embodiments.

Referring to FIG. 13, the display device 100 according to an embodimentmay include a communicator (e.g., including communication circuitry)110, a processor (e.g., including processing circuitry) 120, a display130, and a memory 140.

The communicator 110 according to an embodiment may include variouscommunication circuitry and transmit and receive data or a signal froman external device or an external server under control of the processor120. According to an embodiment, the communicator 110 may communicatewith the mobile device 200 via a wired/wireless network.

The communicator 110 may transmit and receive data or a signal using atleast one of a local area network (LAN), a wide area network (WAN), awireless LAN (WLAN) (for example, Wireless Fidelity (Wi-Fi), WiFiDirect, etc.) Bluetooth, wired Ethernet, infrared (IR), Bluetooth LowEnergy (BLE), ultrasonic waves, or ZigBee. Herein, the communicator 110may include at least one communication module capable of transmittingand receiving data according to communication standards corresponding toan LAN, a WAN, a WLAN (for example, Wi-Fi, WiFi Direct, etc.) Bluetooth,wired Ethernet, IR, BLE, ultrasonic waves, and ZigBee.

The communicator 110 may include a short-range communication moduleincluding various communication circuitry capable of receiving a controlsignal from the control device 50, e.g., an IR communication module,etc. In this case, the communicator 110 may receive a control signalfrom the control device 50.

The communicator 110 according to an embodiment may receive mirroringinformation (mirroring data) and object information corresponding to amirroring screen from the mobile device 200 when the display device 100mirrors a screen output on the mobile device 200. The communicator 110may receive the control signal from the control device 50 and transmituser interaction information included in the control signal to themobile device 200.

The processor 120 according to an embodiment may include variousprocessing circuitry and control the display device 100 overall. Theprocessor 120 may control other components included in the displaydevice 100 to perform a certain operation.

The processor 120 according to an embodiment may execute one or moreprograms stored in the memory 140. The processor 120 may include asingle core, a dual core, a triple core, a quad core, and a core of amultiple thereof. The processor 120 may also include a plurality ofprocessors.

The memory 140 according to an embodiment may store various data,programs, or applications for driving and controlling the display device100.

The programs stored in the memory 140 may include one or moreinstructions. The programs (for example, one or more instructions) orapplications stored in the memory 140 may be executed by the processor120.

The processor 120 according to an embodiment may control the secondscreen corresponding to the first screen output on the mobile device200, based on mirroring data received from the mobile device 200. Theprocessor 120 may obtain navigation map information based on the objectinformation received from the mobile device 200. The processor 120 maymove the indicator on the screen, based on the navigation mapinformation, upon receipt of a user input to move the indicator from thecontrol device 50.

Upon receipt of a control signal including user interaction informationfrom the control device 50, the processor 120 may determine whether totransmit the user interaction information included in the control signalto the mobile device 200 based on whether an operation corresponding toa user interaction causes critical movement, whether the operationcorresponding to the user interaction needs screen changeover, etc. Whenthe operation corresponding to the user interaction does not causecritical movement and does not require screen changeover, the processor120 may perform the operation corresponding to the user interaction. Forexample, a GPU included in the processor 120 may calculate an attributevalue such as coordinates, a form, a size, a color, etc., correspondingto the indicator according to a layout of the screen based on the userinteraction. However, the disclosure is not limited to this example.

The display 130 according to an embodiment may convert an image signal,a data signal, an on-screen display (OSD) signal, a control signal, orthe like, processed by the processor 120, to generate a driving signal.The display 130 may be implemented with a PDP, an LCD, an OLED, aflexible display, or the like, and may also be implemented with athree-dimensional (3D) display. The display 130 may include a touchscreen and thus may be used as an input device as well as an outputdevice.

The display 130 according to an embodiment may display a second screencorresponding to the first screen displayed on the mobile device 200.The display 130 may also display an indicator on the second screen. Thedisplay 130 may move the indicator based on the user interaction anddisplay the indicator.

FIG. 14 is a block diagram illustrating an example configuration of adisplay device according to various embodiments.

A display device 1400 of FIG. 14 may be an embodiment of the displaydevice 100 described with reference to FIGS. 1 and 13.

Referring to FIG. 14, the display device 1400 according to an embodimentmay include a tuner unit (e.g., including a tuner) 1440, a processor(e.g., including processing circuitry) 1410, a display unit (e.g.,including a display) 1420, a communicator (e.g., including communicationcircuitry) 1450, a sensing unit (e.g., including at least one sensor)1430, an inputter/outputter (e.g., including input/output circuitry)1470, a video processor (e.g., including video processing circuitry)1480, an audio processor (e.g., including audio processing circuitry)1485, an audio outputter (e.g., including audio output circuitry) 1460,a memory 1490, and a power source unit (e.g., including a power supply)1495.

The communicator 1450 of FIG. 14 may correspond to the communicator 110of FIG. 13, the processor 1410 of FIG. 14 may correspond to theprocessor 120 of FIG. 13, the display unit 1420 of FIG. 14 maycorrespond to the display 130 of FIG. 13, and the memory 1490 of FIG. 14may correspond to the memory 140 of FIG. 13, respectively, and thus thesame description may not be repeated here.

The tuner unit 1440 according to an embodiment may include a tuner andselect a frequency of a channel the electronic device 1400 is to receivefrom among many electric wave components by tuning the frequency throughamplification, mixing, resonance, or the like with respect to abroadcasting signal received wiredly or wirelessly. The broadcastingsignal may include audio, video, and additional information (forexample, an electronic program guide (EPG)).

The tuner unit 1440 may receive a broadcasting signal from varioussources such as terrestrial broadcasting, cable broadcasting, satellitebroadcasting, Internet broadcasting, and so forth. The tuner unit 1440may receive a broadcasting signal from a source such as analogbroadcasting, digital broadcasting, or the like.

The communicator 1450 according to an embodiment may include variouscommunication circuitry and transmit and receive data or a signal froman external device or a server under control of the processor 1410. Theprocessor 1410 may transmit/receive contents to/from an external deviceconnected through the communicator 1450, download an application fromthe external device, or browse the web.

The communicator 1450 may include a module combination including variouscommunication circuitry in any one or two or more of a WLAN 1451,Bluetooth 1452, and wired Ethernet 1453.

The video processor 1480 may include various video processing circuitryand perform processing on video data received by the display device1400. The video processor 1480 may perform various image processing,such as decoding, scaling, noise filtering, frame rate conversion,resolution conversion, etc., with respect to video data.

The sensing unit 1430 may include at least one sensor and sense a user'svoice, a user's image, or a user's interaction, and may include, forexample, a microphone 1431, a camera unit (e.g., including a camera)1432, and a light receiver (e.g., including light receiving circuitry)1433.

The microphone 1431 may receive an uttered voice of the user. Themicrophone 1431 may convert the received voice into an electric signaland output the electric signal to the processor 1410. The user's voicemay include, for example, a voice corresponding to a menu or a functionof the display device 1400.

The camera unit 1432 may include a camera and receive an image (e.g., acontinuous frame) corresponding to a user's motion including a gesturein a camera recognition range. The processor 1410 may select a menudisplayed on the display device 1400 using a recognition result of areceived motion or perform control corresponding to the recognitionresult of the received motion.

A light receiver 1433 may include various light receiving circuitry andreceive a light signal (including a control signal) received from anexternal control device through a lighting window (not shown) of a bezelof the display unit 1420. The light receiver 1433 may receive a lightsignal corresponding to a user input (e.g., a touch, a press, a touchgesture, a voice, or a motion) from a control device. A control signalmay be extracted from the received optical signal under control of theprocessor 1410.

The inputter/outputter 1470 may include various input/output circuitryand receive video (e.g., moving images, etc.), audio (e.g., a voice,music, etc.), and additional information (e.g., an EPG, etc.) from theoutside of the display device 1400, under control of the processor 1410.The inputter/outputter 1470 may include one of an HDMI port 1471, acomponent jack 1472, a PC port 1473, and a USB port 1474. Theinputter/outputter 1470 may include a combination of the HDMI port 1471,the component jack 1472, the PC port 1473, and the USB port 1474.

The processor 1410 may include various processing circuitry and controlan overall operation of the display device 1400 and a signal flow amonginternal elements of the display device 1400, and process data. Theprocessor 1410 may execute an operating system (OS) and variousapplications stored in the memory 1490, when there is a user input or apreset and stored condition is satisfied.

The processor 1410 may include a random access memory (RAM) that storesa signal or data input from the outside of the display device 1400 or isused as a storage region corresponding to various tasks performed in thedisplay device 1400, a read only memory (ROM) having stored therein acontrol program for controlling the display device 1400, and aprocessor.

The processor 1410 may include a graphics processor (not shown). Thegraphic processor (not shown) may generate a screen including variousobjects such as an icon, an image, a text, etc., using a calculator (notshown) and a rendering unit (not shown). The calculator may calculate anattribute value such as coordinates, shapes, sizes, colors, etc., ofrespective objects based on a layout of the screen using the user'sinput sensed by the sensing unit 1430. The rendering unit may generate ascreen of various layouts including an object based on the attributevalue calculated by the calculation unit. The screen generated by therendering unit may be displayed in a display region of the display unit1420.

The audio processor 1485 may include various audio processing circuitryand process audio data. The audio processor 1485 may perform variousprocessing such as decoding, amplification, noise filtering, etc., onthe audio data. The audio processor 1485 may include a plurality ofaudio processing modules for processing audio corresponding to aplurality of contents.

The audio outputter 1460 may include various audio output circuitry andoutputs audio included in a broadcast signal received through the tunerunit 1440 under control of the processor 1410. The audio outputter 1460may output audio (e.g., voice, sound, etc.) input through thecommunicator 1450 or the inputter/outputter 1470. The audio outputter1460 may output audio stored in the memory 1490 under control of theprocessor 1410. The audio outputter 1460 may include at least one of aspeaker 1461, a headphone output terminal 1462, or a Sony/Phillipsdigital interface (S/PDIF) output terminal 1463. The audio outputter1460 may include a combination of the speaker 1461, the headphone outputterminal 1462, and the S/PDIF output terminal 1463.

The power source unit 1495 may include a power supply and supply power,which is input from an external power source, to the internal elementsof the display device 1400, under control of the processor 1410. Thepower source unit 1495 supplies power, which is output from one or morebatteries (not shown) included in the display device 1400, to theinternal elements, under control of the processor 1410.

The memory 1490 may store various data, programs, or applications fordriving and controlling the display device 1400 under control of theprocessor 1410. The memory 1490 may include a broadcasting receptionmodule, a channel control module, a volume control module, acommunication control module, a voice recognition module, a motionrecognition module, a light reception module, a display control module,an audio control module, an external input control module, a powercontrol module, a power control module of an external device connectedwirelessly (for example, by Bluetooth), a voice database (DB), or amotion DB. Modules and DBs (not shown) of the memory 1490 may beimplemented in the form of software to perform a control function ofbroadcasting reception, a channel control function, a volume controlfunction, a communication control function, a voice recognitionfunction, a motion recognition function, a light reception controlfunction, a power control function, or a power control function of anexternal device connected wirelessly (e.g., by Bluetooth) in the displaydevice 1400. The processor 1410 may perform respective functions usingthe foregoing software stored in the memory 1490.

The block diagrams of the display devices 100 and 1400 shown in FIGS. 13and 14, respectively, may be block diagrams for an embodiment. Elementsof the block diagram may be integrated, added, or omitted depending onthe specifications of the display devices 100 and 1400 implementedactually. When necessary, two or more elements may be integrated intoone element or one element may be divided into two or more elements. Afunction executed in each element (or module) is intended to describeembodiments, and a detailed operation or apparatus thereof does notlimit the scope of the disclosure.

The method of operating the display device according to an embodimentmay be random-access implemented in the form of program commands thatcan be executed through various computer components and recorded in acomputer-readable recording medium. The computer-readable recordingmedium may include a program command, a data file, a data structure andthe like solely or in a combined manner. The program command recorded inthe computer-readable recording medium may be a program commandspecially designed and configured for the present embodiments or aprogram command known to be used by those skilled in the art of thecomputer software field. Examples of the computer-readable recordingmedium may include magnetic media such as hard disk, floppy disk, andmagnetic tape, optical media such as compact disk read only memory(CD-ROM) and digital versatile disk (DVD), magneto-optical media such asfloptical disk, and a hardware device especially configured to store andexecute a program command, such as read only memory (ROM), random accessmemory (RAM) and flash memory, etc. Further, examples of the programcommands include a machine language code made by a complier and ahigh-level language code executable by an interpreter.

The method of operating the display device according to variousembodiments may be included and provided in a computer program product.The computer program product may be traded as a product between a sellerand a buyer.

The computer program product may include a software (S/W) program and anon-transitory computer-readable recording medium in which the S/Wprogram is stored. For example, the computer program product may includea product (e.g., a downloadable application) in the form of an S/Wprogram electronically distributed through a manufacturer or the displaydevice or an electronic market (e.g., Google Play Store or App Store).For the electronic distribution, at least a portion of the S/W programmay be stored in a storage medium or temporarily generated. In thiscase, the storage medium may be a storage medium of a server in themanufacturer or the electronic market or a relay server that temporarilystores the S/W program.

The computer program product may include a storage medium of a server ora storage medium of a client device, in a system including the serverand the client device. Alternatively, when there is a third device(e.g., a smart phone) communicating with the server or the clientdevice, the computer program product may include a storage medium of thethird device. Alternatively, the computer program product may include anS/W program itself, which is transmitted from the server to the clientdevice or the third device or transmitted from the third device toclient device.

In this case, one of the server, the client device, and the third devicemay execute the computer program product to perform the method accordingto the embodiments. Alternatively, two or more of the server, the clientdevice, and the third device may execute the computer program product toexecute the method according to the embodiments in a distributed manner.

For example, a server (e.g., a cloud server or AI server, etc.) mayexecute a computer program product stored in the server to control theclient device communicating with the server to perform the methodaccording to the embodiments.

As the display device according to an embodiment receives the objectinformation corresponding to a mirroring screen and controls movement ofthe indicator based on the object information and the user interaction,the user watching the mirroring screen displayed on the display devicemay easily manipulate the mirroring screen using the control device.

The user of the display device according to an embodiment maycontinuously enjoy the “lean-back experience” without being disturbed,when the user watches the mirroring screen displayed on the displaydevice.

While the disclosure has been illustrated and described with referenceto various example embodiments, it will be understood that the variousexample embodiments are intended to be illustrative, not limiting. Itwill be further understood by one skilled in the art that variouschanges in form and detail may be made without departing from the truespirit and full scope of the disclosure including the appended claimsand their equivalents.

What is claimed is:
 1. A display device comprising: a display; acommunicator comprising communication circuitry configured tocommunicate with a mobile device; a memory storing one or moreinstructions; and a processor configured to execute the one or moreinstructions stored in the memory to: based on the display deviceperforming a screen mirroring with respect to a first screen of a mobiledevice, control the display to display a second screen corresponding tothe first screen and an indicator; control the communicator to receiveobject information regarding one or more objects included in the firstscreen from the mobile device; generate navigation map information ofthe one or more objects based on the object information; and in responseto an input for moving the indicator, control movement of the indicatorbased on the navigation map information.
 2. The display device of claim1, wherein the object information comprises at least one of positioninformation of the one or more objects, information about an objectadjoining the one or more objects, timestamp information, or informationregarding whether the one or more objects are associated with criticalmovement.
 3. The display device of claim 1, wherein the processor isfurther configured to execute the one or more instructions to: allocatea number to each of the one or more objects included in the secondscreen and control the display to display the allocated number.
 4. Thedisplay device of claim 3, wherein the processor is further configuredto execute the one or more instructions to: in response to an input forselecting the number, move the indicator to an object to which theselected number is allocated.
 5. The display device of claim 1, whereinthe processor is further configured to execute the one or moreinstructions to: determine, based on the object information and thenavigation map information, whether the input for moving the indicatoris regarding an object associated with critical movement; and controlthe communicator to transmit information regarding the input for movingthe indicator and the object to the mobile device based on the input formoving the indicator being regarding the object associated with criticalmovement.
 6. The display device of claim 5, wherein the input for movingthe indicator being regarding the object associated with criticalmovement comprises an input for moving the indicator to an objectcausing a page changeover operation among the one or more objects. 7.The display device of claim 5, wherein the processor is furtherconfigured to execute the one or more instructions to: based on a screenof the mobile device being changed from the first screen to a thirdscreen as the mobile device performs an operation corresponding to theinformation regarding the input for moving the indicator and the object,control the display to display a fourth screen corresponding to thethird screen by mirroring the third screen.
 8. The display device ofclaim 5, wherein the processor is further configured to execute the oneor more instructions to: perform an operation corresponding to the inputfor moving the indicator based on the input for moving the indicatorbeing not regarding the object associated with critical movement.
 9. Amethod of operating a display device, the method comprising: based onthe display device performing a screen mirroring with respect to a firstscreen of a mobile device, displaying a second screen corresponding tothe first screen and an indicator; receiving object informationregarding one or more objects included in the first screen from themobile device; generating navigation map information of the one or moreobjects based on the object information; and in response to an input formoving the indicator, controlling movement of the indicator based on thenavigation map information.
 10. The method of claim 9, wherein theobject information comprises at least one of position information of theone or more objects, information about an object adjoining the one ormore objects, timestamp information, or information regarding whetherthe one or more objects are associated with critical movement.
 11. Themethod of claim 9, further comprising allocating a number to each of theone or more objects included in the second screen and displaying theallocated number.
 12. The method of claim 11, wherein the controlling ofthe movement of the indicator comprises, in response to an input forselecting the number, moving the indicator to an object to which theselected number is allocated.
 13. The method of claim 9, wherein thecontrolling of the movement of the indicator comprises controlling themovement of the indicator based on a control signal including userinteraction information regarding moving the indicator and thenavigation map information.
 14. The method of claim 13, furthercomprising: determining, based on the object information and thenavigation map information, whether the input for moving the indicatoris regarding an object associated with critical movement; andtransmitting information regarding the input for moving the indicatorand the object to the mobile device based on the input for moving theindicator being regarding the object associated with critical movement.15. The method of claim 14, wherein the input for moving the indicatorbeing regarding the object associated with critical movement comprisesan input for moving the indicator to an object that causes a pagechangeover operation among the one or more objects.
 16. The method ofclaim 14, further comprising displaying a fourth screen corresponding toa third screen by mirroring the third screen based on a screen of themobile device being changed from the first screen to the third screen asthe mobile device performs an operation corresponding to the informationregarding the input for moving the indicator and the object.
 17. Theoperating method of claim 14, further comprising performing an operationcorresponding to the input for moving the indicator based on the inputfor moving the indicator being not regarding the object associated withcritical movement.
 18. A non-transitory computer-readable recordingmedium having recorded thereon a program, which when executed by aprocessor included in a display device, causes the display device toperform operations including: based on the display device performing ascreen mirroring with respect to a first screen of a mobile device,displaying a second screen corresponding to the first screen and anindicator; receiving object information regarding one or more objectsincluded in the first screen from the mobile device; generatingnavigation map information of the one or more objects based on theobject information; and in response to an input for moving theindicator, controlling movement of the indicator based on the navigationmap information.